Bring the cloud (cloudbridge) object store to parity with the boto3 store#23098
Bring the cloud (cloudbridge) object store to parity with the boto3 store#23098nuwang wants to merge 20 commits into
Conversation
…entials The aws branch of Cloud.parse_xml checked `"region" in auth_element` — membership on an XML Element iterates child elements, so the check was always false and a configured region was silently dropped (the downstream connection code at _get_connection already handled a region correctly when present). The google branch called os.path.isfile() on the credentials_file value before checking it for None, so a missing attribute produced a TypeError instead of the intended missing-configuration error.
Both S3 stores set the `cloud = True` class attribute (surfaced through badge serialization); the cloudbridge store left it at the ConcreteObjectStore default of False.
Both push paths did an objects.get() to test existence and then a second get() to fetch the same object for upload. Get-or-create once and upload on the returned object.
A new <transfer> block (multipart_threshold, multipart_chunksize, max_concurrency - byte sizes, boto3-store naming) maps onto cloudbridge's per-call UploadConfig. multipart_chunksize below the 5 MiB provider minimum is rejected at construction time rather than failing on first large upload. Only upload tuning is exposed: cloudbridge has no download-side transfer configuration (downloads stream in fixed chunks).
upload()/upload_from_file() now receive the transfer options parsed from the <transfer> block; with nothing configured they receive config=None and cloudbridge's own multipart defaults apply. cloudbridge handles the multipart mechanics transparently above the threshold.
The enable_direct_download plumbing is generic (ConcreteObjectStore dispatches to _get_object_url), but the cloud store's _get_object_url accepted content_disposition/content_type and dropped them. Forward both to cloudbridge's generate_url so redirected downloads carry the forced filename and MIME type (requires cloudbridge >= 4.2.1; on OpenStack Swift only the filename portion is honored).
…tore The store surfaced only a fraction of what each cloudbridge provider accepts; several missing options are required in real deployments: - aws: session_token (STS/temporary credentials) on <auth>, and a <connection> block with endpoint_url (S3-compatible endpoints such as MinIO/Ceph RGW - the scheme conveys http/https, so no is_secure, matching the boto3 store), validate_certs and signature_version. - azure: storage_account and resource_group - without these cloudbridge synthesizes a UUID-based storage-account name, so an existing storage account was unreachable. Also region and access_token (with a token the service-principal quartet becomes optional). - google: credentials_dict as an inline alternative to credentials_file (YAML configurations; exactly one of the two is required) and region. Compute-only cloudbridge options (zones, VM settings, ec2_*) are deliberately not exposed. Unset options are omitted from the provider config so cloudbridge's own env-var fallbacks still apply.
Native Swift object storage via cloudbridge's OpenStack provider, giving swift/generic_s3 deployments a migration target ahead of the legacy boto2 store's deprecation. Authentication supports Keystone v3 password auth (username/password/project_name) or an application credential, plus auth_url (required), region and domain names. Requires the cloudbridge[openstack] extras in Galaxy's environment, matching how the azure/google providers expect their SDKs.
verify_caching_object_store_functionality's "bigger file" check never actually uploaded the big file - it wrote big_file.bytes but passed hello_path to update_from_file, and asserted nothing. Upload the right file and assert existence and size. A new verify_big_file_storage_roundtrip helper stores a file large enough to cross the multipart threshold and verifies a bit-for-bit SHA256 roundtrip through the remote store after cache resets. Two new env-gated tests exercise it with a 12 MiB file over a 5 MiB threshold/chunksize: against real S3 (cloudbridge maps the config onto boto3's TransferConfig) and against real Swift (cloudbridge's generic clone-pool multipart driver, gated on GALAXY_TEST_OS_*).
Runs the cloudbridge store (provider=aws + endpoint_url) against MinIO in docker - no cloud credentials needed - with the multipart threshold at the 5 MiB provider minimum, so the tool-test datasets and an explicit 12 MiB roundtrip exercise the multipart upload path in CI. Verified locally against MinIO: bucket auto-creation, the full caching object store lifecycle including presigned-URL download, and the 12 MiB multipart roundtrip (SHA256-checked).
Covers direct download, STS session tokens, S3-compatible endpoints, multipart transfer tuning, Azure storage_account/resource_group/ access_token, GCP inline credentials, and the new openstack provider.
4.2.1 supports Python 3.10+ (so the split 3.2.0/4.0.x pins collapse to one) and provides the multipart UploadConfig and generate_url response-header parameters the cloud object store now uses. Pins regenerated with ./lib/galaxy/dependencies/update.sh -p cloudbridge, which also picked up patch-level drift in a few unrelated packages. Also add cloudbridge to the objectstore package's test extra.
…tore Downloads now go through cloudbridge's download_to_file, which fetches objects above the transfer threshold as ranged reads in parallel - mirroring what boto3's TransferManager gives the boto3 store - instead of a single-connection stream. The axel fallback is dropped: parallel downloads are now native, and the boto3 store never used axel either. The <transfer> options gain optional upload_/download_ prefixes to tune one direction only (bare keys apply to both), matching the boto3 store's convention, and are validated accordingly: the 5 MiB part floor is an upload constraint, so download chunk sizes below it are legal. cloudbridge 4.3.0 also renames UploadConfig to TransferConfig now that the same knobs drive both directions.
Adds "cloud" to ObjectStoreTemplateType with pydantic template and configuration models covering the store's full surface: provider (aws/azure/google/openstack), a flat per-provider auth model (which fields apply depends on the provider - the store validates the combination at construction), bucket, connection and transfer options. Includes a production catalog example for an S3 bucket via CloudBridge and the regenerated client API schema plus the template type's icon/message entry in the client.
cloudbridge's objects.list() returns a single page (default 50 items), and the store iterated only that first page when deleting all objects under a prefix and when downloading a directory into the cache - a dataset with more extra files than the page size was deleted or downloaded incompletely. Use objects.iter(), which pages through the complete result set on every provider. The directory existence check keeps list(): any match at all puts at least one object on the first page.
The referenced issue was a typo (CloudVE/cloudbridge#135 is unrelated; the credential-check discussion is galaxyproject#120). The reasoning still holds - cloudbridge's authenticate() is compute-scoped, so a least-privilege bucket-only credential would fail it - and since the bucket lookup in _initialize now exercises the credentials with a storage-scoped call, bad credentials fail fast at startup anyway.
…store too The object store config becomes a class attribute so the cloud (cloudbridge) store runs the same five web-layer scenarios the boto3 store does, against its own MinIO container: /download 302s to a presigned URL carrying the response content disposition, the URL is fetchable directly from the store with no cache pull, HEAD serves metadata without a redirect, and /display keeps streaming. Also pin down enable_direct_download on the cloud store's to_model serialization, matching the boto3 parse test.
ObjectStoreTypeSpan.vue indexes an exhaustive record over the template type enum, so it needs a message for the new cloud type. The paging fake's SimpleNamespace provider gets a cast for mypy - only the result-limit config is consulted by ClientPagedResultList.
The paragraph predates this branch: the cloud store now has parallel multipart transfers in both directions with per-direction tuning, and handles S3-compatible endpoints via endpoint_url.
f48bcb0 to
fc6dcbd
Compare
| Stores Galaxy datasets in an Amazon S3 bucket through the | ||
| provider-agnostic cloud object store. Files larger than the multipart | ||
| threshold transfer as parts in parallel, in both directions. |
There was a problem hiding this comment.
This is a user facing file - to keep it inline with the other examples I would not include such technical language.
jmchilton
left a comment
There was a problem hiding this comment.
To bring it to parity I'd also recommend updating docs/ around this. All the other templates I think have admin-facing documentation, schema docs, etc... But I don't think it is strictly required. Also just a warning - the template objects that can be used to defined user-defined object stores need to essentially maintain backward compatibility indefinitely because those configs are defined in the database. Therefore, I tried to be a little conservative about what object stores and what configs I expose in those places based on that (admittedly I may have not used such restraint with the boto3 implementation though). So I just want to make sure you're comfortable with cloudbridge supporting this interface defined in that model layer indefinitely. I don't see any obvious problems - just letting you know that I'm uneasy every time this interface expands and this is a substantial expansion.
Brings
lib/galaxy/objectstore/cloud.pyto functional parity with the modern boto3 store, making it a viable multi-cloud successor ahead of eventually deprecating the legacy boto2s3.pystore. Builds on cloudbridge 4.3.0, which added cross-provider multipart uploads (CloudVE/cloudbridge#333), a Python 3.10 floor (#338), response-header overrides on signed URLs (#339), and ranged parallel downloads (#340).What's included (one commit per change, tests first in each)
Bug fixes
regionon<auth>was silently dropped ("region" in auth_elementiterates child elements, never attributes)credentials_fileraisedTypeErrorinstead of the missing-configuration errorobjects.get()per uploadcloud = True(badge serialization)list()returns one page, default 50 items) — delete-all and directory-into-cache handled datasets with many extra files incompletely; they now page through the full result set viaiter()verify_caching_object_store_functionality's "bigger file" was written but never uploaded (and nothing was asserted)Parity features
<transfer>block (multipart_threshold,multipart_chunksize,max_concurrency— boto3-store naming, with optionalupload_/download_prefixes to tune one direction) mapped to cloudbridge's per-callTransferConfig. Upload chunk sizes below the 5 MiB provider minimum are rejected at construction (downloads have no minimum)._download_tonow uses cloudbridge'sdownload_to_file, which fetches large objects as parallel ranged reads — mirroring what boto3's TransferManager gives the boto3 store. The axel fallback is dropped (parallel downloads are native now; the boto3 store never used axel either).enable_direct_downloadnow works for the cloud store —content_disposition/content_typeare forwarded togenerate_url(full support on AWS/Azure/GCP; Swift honors the filename portion via tempurl).session_token+<connection endpoint_url= validate_certs= signature_version=>(S3-compatible endpoints; the scheme conveys http/https, matching the boto3 store's dropping ofis_secure); Azurestorage_account/resource_group(previously unreachable — cloudbridge synthesizes a UUID-based storage-account name without them),region,access_token; GCP inlinecredentials_dict+region.provider: openstack: native Swift via Keystone v3 (password or application-credential auth), giving swift deployments a migration target."cloud"is now anObjectStoreTemplateTypewith full pydantic template/configuration models (provider, per-provider auth, bucket, connection, transfer), a production catalog example, and the client schema/type entries.Tests
GALAXY_TEST_OS_*), both exercising a SHA256-checked big-file roundtriptest/integration/objectstore/test_cloud_objectstore.py) — deterministic multipart coverage in CI with no cloud credentialstest_direct_download_redirect.py) now also runs against the cloud store (same five web-layer scenarios, own MinIO container)Dependencies:
cloudbridge>=4.3.0; the split3.2.0/4.0.xpins collapse to one now that cloudbridge supports Python 3.10+. Pin changes are limited to cloudbridge itself.How to test
pytest test/unit/objectstore/test_objectstore.py -k cloud(63 tests in the file pass overall)pytest test/integration/objectstore/test_cloud_objectstore.py(docker)GALAXY_TEST_AWS_* pytest test/unit/objectstore/test_objectstore.py -k cloudbridge_store_multipartDeferred follow-ups
s3.pydeprecation announcement + migration docs (aws_s3 → boto3/cloud; swift/generic_s3 → boto3+endpoint_url or cloud+openstack)